A Global Gmres/multi-grid Scheme for an Adaptive Cartesian/quad Grid Flow Solver on Distributed Memory Machines

A global multi-grid/GMRES solution methodology on distributed memory machines is successfully developed in this study. To preserve the effectiveness of the multigrid scheme, the grid partitioning is based on the communication graph of the coarsest grid, so that all levels of the multi-grids are located in the same zone (processor). Each node of the graph is weighted with the total number of the finest grid cells contained in the coarsest grid cell corresponding to the node. This weighted graph is then partitioned with either a Recursive Spectral Bisection (RSB) algorithm or a Recursive Coordinate Bisection (RGB) method to balance the load on the finest grid. With this type of domain decomposition, there is no communication overhead with the prolongation and restriction operations in the multi-grid algorithm. A global GMRES scheme with a multi-grid preconditioner is implemented to drive flows to steady state. The GMRES algorithm is completely parallelizable. Data communication between different zones is provided through a Message Passing Interface (MPI) library, MPICH from Argonne National Laboratory. In addition, a novel communication and computation overlap (CCO) procedure is implemented to further reduce communication time by 10-20%. The performance of the overall algorithm was assessed on the IBM SP2 parallel computer at Maui High Performance Computing Center (MHPCC). Excellent scalability of the overall methodology was demonstrated.